AutocompleteTemplate: Use modelfile template for granite3 #3314
Conversation
granite3 is currently using the "hole filler" template since it lacks FIM support. This template is getting sent to ollama raw, so the template from the model file is not getting applied to it. This commit ensures the proper format is fed to the model, by moving the hole filler system message to be a global constant, and using it in the existing holeFillerTemplate and a new granteHoleFillerTemplate that also applies the appropiate model specific formatting.
✅ Deploy Preview for continuedev ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
this is marked draft because i haven't tested it yet, so haven't confirmed it doesn't regress results. I just sketched it out a few days ago after noticing the problem when looking at ollama debug output. I will test eventually, but for now, I just want to get it off my machine so I don't lose track of it. |
| ) { | ||
| return holeFillerTemplate; | ||
| } | ||
|
|
||
| if (lowerCaseModel.includes("granite3")) { |
There was a problem hiding this comment.
lowerCaseModel.includes("granite3") && !lowerCaseModel.includes("-code")
future proofing the next models that'll support FIM
There was a problem hiding this comment.
given -code models won't be released for 3.2, you might want to use :
lowerCaseModel.match(/^granite3(?:\.1)?-/)
|
@halfline have you had any time to take a second look at this? |
not yet and the latest granite release has a different template. will get back to this soon |
|
I think it may make sense to close this PR for now, especially as we're trying to clean up a large backlog of PRs. @halfline if you get the chance to come back to this let me know and we're of course happy to re-open! |
granite3 is currently using the "hole filler" template since it lacks FIM support. This template is getting sent to ollama raw, so the template from the model file is not getting applied to it.
This commit ensures the proper format is fed to the model, by moving the hole filler system message to be a global constant, and using it in the existing holeFillerTemplate and a new granteHoleFillerTemplate that also applies the appropiate model specific formatting.